Skip to content

Feat(sqlglotrs): match the Python implementation of __repr__ for tokens #5172

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
georgesittas merged 1 commit into from
Jun 5, 2025
Merged

Feat(sqlglotrs): match the Python implementation of __repr__ for tokens #5172

georgesittas merged 1 commit into from
Jun 5, 2025

Conversation

@georgesittas
Copy link
Collaborator

@georgesittas georgesittas commented Jun 5, 2025 •
edited
Loading

Behavior in main today when using the Rust tokenizer:

>>> import sqlglot
>>> sqlglot.tokenize("foo")
[<builtins.Token object at 0x105d7adf0>]

Behavior in this PR:

>>> import sqlglot
>>> sqlglot.tokenize("foo")
[<Token token_type: TokenType.VAR, text: foo, line: 1, col: 3, start: 0, end: 2, comments: []>]

Behavior in both main and this PR when using the Python tokenizer:

>>> import sqlglot
>>> sqlglot.tokenize("foo")
[<Token token_type: TokenType.VAR, text: foo, line: 1, col: 3, start: 0, end: 2, comments: []>]

@georgesittas georgesittas requested a review from benfdking June 5, 2025 12:48
@github-actions
Copy link
Contributor

github-actions bot commented Jun 5, 2025

Benchmark for dab9b7d

Click to view benchmark
Test Base PR %
long 215.6±3.08µs 215.4±5.51µs -0.09%

@georgesittas georgesittas merged commit 63f9cb4 into main Jun 5, 2025
8 checks passed
@georgesittas georgesittas deleted the jo/improve_sqlglotrs_token_repr branch June 5, 2025 13:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@benfdking benfdking Awaiting requested review from benfdking

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@georgesittas